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Remote Operations and Ground Control Centers 

Abstract: The Payload Operations Integration Center (POIC) at the Marshall Space Flight Center 
supports the International Space Station (ISS) through remote interfaces around the world. The POIC 
was originally designed as a gateway to space for remote facilities; ranging from an individual user to a 
full-scale multi-user environment. This achievement was accomplished while meeting program 
requirements and accommodating the injection of modem technology on an ongoing basis to ensure cost 
effective operations. 

This paper will discuss the open POIC architecture developed to support similar and dissimilar remote 
operations centers. It will include technologies, protocols, and compromises which on a day to day basis 
support ongoing operations. Additional areas covered include centralized management of shared 
resources and methods utilized to provide highly available and restricted resources to remote users. 
Finally, the effort of coordinating the actions of participants will be discussed. 
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Remote Operations and Ground Control Centers 


Introduction 

The goal of this paper is to build a case that remote operation is not only a cost effective but viable 
method of conducting space operations. This includes the ability for users to conduct science from 
wherever it is comfortable and efficient. To fully cover this position, a detailed discussion of remote 
operations will be given to include the following areas: 

1 . Aspects of remote operations 

2. The ISS Payload Operations Integration Center (POIC) remote operations model 

3. Enabling items of the POIC architecture 

4. Remote Operations Objectives 

5. Remote Operations Options and Services 

6. Lessons Learned 

7. Benefits for Future Programs 

At the conclusion of this paper, the reader should feel confident that remote operations have been 
successful in pursuit of operations by a diverse community. This will be reinforced by a long and fruitful 
history and a bright future. The future of remote operations is ensured by a rich suite of capabilities 
supported by iow cost and pervasive technology. 

Introduction to remote operations 

Remote operations, when implemented properly, allows an operator from virtually anywhere in the world to 
control and maintain payload operations. The user is capable of accomplishing most objectives to include 
normal health and safety activities as well as the conduct of science. With the proper collaborative tools, 
this need not be limited to rote procedure execution. New activities and directions can be orchestrated 
and executed efficiently. In fact a remote user or group can develop their own operational model making 
use of expertise which would be unavailable due to cost or prohibitions if deployed in a traditional control 
room setting. 

Remote operations capabilities are now available inexpensively through the use of several enabling 
technologies and trends. Specifically, Moore’s Law and capitalism has spurred the development of low 
cost, high performance compute platforms at reduced prices; e.g. Intel-like platforms. Bandwidth has 
been on the increase to the desktop through the use of switch-based networks. In addition, users have 
ubiquitous internet access at much higher rates than ever before. Virtual Private Networks (VPN) 
provides secure connections between users and the Huntsville Operations Support Center (HOSC) 
resources that houses the POIC. The use of VPNs has not only provided secure communications but has 
been incorporated with minimal infrastructure changes, has simplified to the overall architecture, and 
reduced the cost of HOSC services. Finally, technology is now available for low cost systems to be more 
highly available than ever before. Therefore with the proper facilitization, inexpensive and resilient 
systems can be deployed with little or no downtime. 

These enabling technologies have been supported and promoted by several trends in the Information 
Technology (IT) culture. First, commodity platforms and the operating systems executing on them 
(Windows™ and Linux) are becoming extremely reliable. In many cases it is comparable to proprietary 
systems by major vendors. Many product developers are supporting and incorporating a wide range of 
standards into their products. The HOSC has made use of these standards to promote interoperability 
with its users and to reduce the overall cost of deploying systems. Such standards as IPSec, CORBA, 
and POSIX insure that independent computing environments from different vendors are compatible. 
Finally, two of the most notable trends are the Free Software and Open Source movements. By providing 
operating systems such as Linux and associated toolsets, attractive alternatives to proprietary systems 
are available. This engenders competition that is essential to reducing cost and increasing 
interoperability. 
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Payload Operations Integration Center (POIC) 
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’ Any Remote Payload Option can utilize the EHS Web interfaces 


Figure 1 -Remote Payload Options 


Other items make remote operations not only critical but also desirable. For instance, tighter fiscal 
policies forced users and the ISS program to search for inexpensive ways to accomplish science. As the 
number of experiments increases, the associated manpower grows. Most universities and research 
institutions are accessible via the internet. Therefore Collaboration between disciplines and dissemination 
of science requires that many of the HOSC applications be internet enabled. Finally, users want to do 
their work at home in a familiar environment close to their colleagues. 

For these reasons, the HOSC became a facility where remote operations is standard, not an exception. 
The HOSC can support users local to its facility but that is not preferred. Several remote payload options 
are available depending on the needs of the user. As shown in Figure 1, they include the Telescience 
Resource Kit (TreK), remotely interfaced Ground Support Equipment (GSE), Enhance HOSC System 
Personal Computer (EPC), Internet Voice Distribution System (IVoDS), Enhanced HOSC System Web 
(EHS Web), or an independent copy of the EHS system. 

TReK is a Flight Projects developed software and COTS suite of tools to process telemetry and command 
to the ISS system. Designed for the Windows operating system, it is currently supported on Windows 
2000™. Services TReK provides are ground support for telemetry processing and commanding and an 
interface to the POIC for access to the ISS. 

GSE interfaces are provided as detailed in the POIC to Generic User Interface Definition Document 
(PGUIDD SSP-50305). This document defines allowable interfaces to the HOSC. The interfaces are 
defined as a set of protocols and standards which given the proper authorization allows a users 
independence of platform type. Therefore a user’s GSE can be anything from a mainframe to a PC if the 
platform supports internet protocols. 

Due to increased cost of proprietary systems and increased performance of the personal computer, Flight 
Projects and the HOSC has developed a PC base client for its POIC ground system called EPC. The 
EPC allows the a user to access HOSC EHS services. A user may be remote or local to the HOSC. 
Hosted on a Windows 2000™ platform, EPC provides telemetry processing and commanding capabilities 
to include scripting and computation execution over a secure interface. 
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IvoDS is a Flight Projects developed Voice Over Internet Protocol (VOIP) system allowing remote users 
the ability to participate in operational voice loops through the use of their PC without traditional 
telecommunication equipment. This capability allows any authorized user to monitor up to eight (8) loops 
or conference networks simultaneously and talk on one (1) network. This is accomplished over a secure 
network connection. 

The HOSC provides web services for users to access various web enabled POIC capabilities from a JAVA 
compliant Windows 2000™ platform. Through a web browser and the use of a JAVA plugin, many POIC 
remote functions are accessible to include the control of realtime telemetry, database definitions, and 
collaborative resources of the Payload Information Management System. These services are delivered 
via highly reliable, available, and secure POIC web servers. 

Finally, the HOSC can provide a remote copy of itself which can execute on low cost commodity server 
platforms running the Linux O/S with Windows 2000™ clients. Among the benefits provided by this option 
is independent operation of a complete operations center or the ability to interoperate with the HOSC. 

Each EHS copy is capable of both local and remote operations. 

Introduction to the POtC 


Marshall Space Flight Center has a long and distinguished history of support through remote means. In 
1961 MSFC installed facsimile services to exchange data and documents with the Air Force station at 
Cape Canaveral to expedite the Saturn development. In 1965, Von Braun established the Mission 
Operations Office (MOO) and the MOO created the Launch Information Exchange Facility (LIEF). Soon 
after the HOSC was created as part of the LIEF. The HOSC evaluated launch data from KSC and flight 
data from JSC remotely. The LIEF was characterized as the “ all hearing center during a Saturn launch”. 


Throughout the 70s, the HOSC supported NASA. In 1973, the HOSC was online and supporting Skylab. 
Its special troubleshooting team supported from problem detection to resolution. When the Apollo-Soyuz 
was active, the HOSC was online and supporting as well. 
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Figure 2- Remote Users of the HOSC 


Throughout the 80s and 90s, the HOSC provide continuous support to STS launches and Spacelab 
missions. Users from all around the world were involved. Most support started local but slowly evolved. 
Eventually only a tele-presence was required for many payload investigators. 
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The HOSC’s missions today include ISS payload operations, STS launch support, Chandra X-ray 
observatory ground system development, and emergency backup for the ISS Mission Control Center in 
Houston (MCC-H). Specifically, ISS operations support a diverse user base of payload operations and the 
HOSC is the dissemination point for all ISS science data. All payloads are supported simultaneously 
whether they are facility class or individual experiments. 

Therefore the HOSC is accessible from around the world as shown in Figure 2 by users requiring realtime 
and support services. The services can be as diverse as voice, video, telemetry, database, or 
commanding; either in realtime or from archived data. A user can command, manage payloads, and 
process telemetry for health, status, or science. At the HOSC, payload system managers (Cadre) support 
the overall integration of payload operations. The Cadre ensures that mission success and safety are 
fulfilled for all users and interfaces directly with the astronauts and MCC-H personnel as advocates for the 
investigators. 

PO*C Architecture 

The POIC architecture has evolved over many years to provide a cost effective, high available system rich 
with remote operations capabilities. The HOSC has been designed to provide support from single users 
up to large facilities. As such the HOSC offers services through three (3) primary systems: 


• Enhanced HOSC System (EHS) 

• Payload Data Services System (PDSS) 

• Payload Planning System (PPS) 

A number of auxiliary services are available as well: 

• Traditional Voice Distribution System 

• Internet Voice Distribution System (IVoDS) 

• Video Distribution System 

• Telescience Resource Kit (TReK) 


Through these systems the POIC provides an extensive suite of Ground System capabilities to support 
payload operations needs. These capabilities have evolved over many years of local and remote user 
support in the HOSC. All capabilities provided to local HOSC users are also available to our remote user 
community. These capabilities are available to remote users through the various options defined earlier in 
this paper. The suite of capabilities can be divided into several key support areas: 

• Data Acquisition and Distribution Services 

• Database Services 

• Telemetry Services 

• Command Services 

• Payload Information Management System Services 

Data Acquisition and Distribution Services provides the capability to acquire telemetry data from multiple 
sources, generate data quality information, provide highly available data storage and management, 
provides the capability to playback any stored telemetry data, and provides a standard delivery method for 
science data to all local and remote users. 

Database Services includes conversion of the ISS Payload Data Library (PDL) input database into the 
HOSC Command and Telemetry database formats. The POIC systems can process all data and 
commands defined in the POIC databases. Database services also include a centralized Database 
Change Request System that allows local and remote users to easily identify required changes to the 
baseline database. 

Telemetry Services include the capability to log, manage retrieve and playback processed data. It 
includes the capability to use a simple point and click user interface to build graphical, textual and tabular 
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displays in realtime. No programming experience is required to create and validate user displays. A 
display can be created in validated in a matter of minutes. Telemetry services also provide the capability 
for Near Realtime (NRT) processing of stored telemetry data. This includes the capability to request a 
selection of parameters over a specified time period and receive the resulting data in file, display or plot 
formats. Telemetry Services includes the capability for a remote user to define the telemetry parameters 
required and the POIC systems will create a realtime data packet and distribute the data to the remote 
user. This prevents the remote user from having to ingest the entire telemetry stream. Other capabilities 
include an extensive Exception Monitoring capability and the ability for users to create and run 
computations that are driven by realtime telemetry. 

Command Services provides the user with the capability to view, modify, initiate and status commands. It 
also provides the unique controls and security functions required to safely execute hazardous and critical 
commands. Command services also provide a programmatic interface to remote users for remote user 
commanding. The command system has been extended to prove an extensive Scripting capability that 
allows a user to define individual scripts to perform routine functions. The script can be as simple and 
perform a single function or they can be quite complex and include nested scripts to perform various 
functions according to a user timeline or user input. These functions may include starting displays or 
sending commands based on downlink telemetry values or other conditions determined by the user. 
Command Services also provides the capability for a user to track command progress in realtime, create 
command delogs of specified periods of time and create long term command histories. 

Payload Information Management System (PIMS) Services provide a workflow engine with user 
notification and status updates included in an online Operations Change Request system. PIMS provides 
a data vault for the storage and management of operational procedures, documents and OCRs. PIMS 
also acts as an operational hub for payload users to interchange flight items with the Cadre. 

Systems in the POIC are partitioned into Private and Public domains. The Private domain is secured by 
an advanced firewall with stateful inspection of protocols. The firewall is IPSec 1.0b compliant and is 
supported by a knowledgeable and responsive vendor. Public domain traffic is secured by router access 
lists and firewalls as well. Traffic between Private and Public domains is restricted to only those protocols 
required to support ISS operations. Behind this are the Mission LANs with the most strict access 
requirements. 

As shown in Figure 3, the POIC is composed of three primary layers or tiers; Tier 1 for clients, Tier 2 with 
Interface servers, and on Tier 3 reside the application servers with provide services. A fourth 
infrastructure layer supports common functions; e.g DNS, LDAP. Each tier fills a unique role. 

The client systems of Tier 1 can be found internally or remote from the POIC. External clients must have 
communications by VPN and be isolated from promiscuous access. Internal clients are located on 
switched networks and are rigorously managed. A client may host web, x-server, or native applications to 
provide a complete ground system solution for ISS payload users. Users are allowed access to the 
second-tier systems but are not allowed on the third tier. 

The POIC developed clients have been developed on low cost generic platforms with a Windows 2000™ 
O/S. Software components were developed using modern techniques and languages such as Visual 
Basic, JAVA, C, and C++. A user can process realtime data or command a payload while doing everyday 
work using Microsoft Office or another common COTS product on the same PC. 

On tier 2 resides three types of interface servers. Web servers provide payload services to local and 
remote users. These servers are outside of the mission systems networks but behind one side of a mutli- 
sided firewall. The web servers are sacrificial and no persistent data are stored on them. Additionally, 
client passwords and privileges are stored on mission systems. Request for access is relayed to the 
mission systems application servers. Web users are restricted to web servers based on the users’ IP 
address. 

The second type of interface server is the Gateway or Login server. These servers act as the gateways of 
the clients for non-web based services that require user authentication and privacy. These platforms 
provide processed data or X-client services to authenticated clients. Like web servers, gateway servers 
are sacrificial and no persistent data are stored on them. Gateway users are restricted based on the 
users’ IP address. 
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Tier 2 servers have been developed on commodity Intel™ platforms. The POIC is completing the 
rehosting and migration of software to these low cost platforms. The operating system selected was 
Linux™ and a specific vendor was chosen to provide product updates. As a result, the portage cost is 
offset by the net cost of new platforms that provide better performance than equivalent proprietary 
platforms. 

The final interface server type is the Remote Interface System (RIS) server. This server provides data to 
multiple remote users with only the requesting remote user authenticated. Like other Tier 2 servers, RIS 
servers are sacrificial and no persistent data are stored on them. 



Figure 3- POIC Architecture 


The application servers are mission critical devices and make up the third Tier. They encompass a broad 
range of services to include ISS drop boxes, ISS command services, telemetry archive, project database, 
telemetry and science data distribution, long and short term plan activities, exception monitoring, and an 
advanced information management system. These systems are considered inviolate and protected at all 
cost not only for the data they manage but for the interfaces they respond to. Like the Tier 2 servers, the 
application servers are hosted on low cost commodity platforms. 

Remote Operations Objectives 

Two sets of objectives exist; the user’s and the system's. A typical user has basic objectives that allow for 
easy operations concept integration. Users primarily have the following objectives: 

• Simple, tailored, convenient, and stable interfaces for voice, video, data, and command services 

• Support through all mission phases (i.e., pre-mission planning, mission preparation testing, real-time 
support, post-mission analysis) 

• Getting the right information (data) to the right people at the right time 

• Being provided a “just right” solution. “Don’t provide a luxury car when all I need is a compact.” 

• All data only when needed 

The system has another set of objectives. These objectives are supportive of users but sometimes 
appear to be in conflict due to constraints imposed on the users: 
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• Provide responsive support services (i.e., mission integration, customer support, training, etc.) 

• Insure the integrity and availability of data even under near catastrophic events (e.g provide RAID, 
backups, and downtime for maintenance activities) 

• Provide a secure environment and communications channels 

• Provide a cost effective solution for ongoing activities 

• Provide facilitization of users and equipment 

Remote Operations Options and Services 

When evaluating remote operations options, a user can develop their products to meet the program 
interfaces of the control center, use the control center’s capabilities, or a combination of the two. 

A user may decide to develop all of his tools and only use the baselined program interfaces provided by 
the control center. In choosing this option, the user needs to consider the lifecycle cost of the 
development of these tools, the extra testing required to certify the product, the training requirements, and 
the flight certification requirements. One disadvantage to this option is that the expense incurred 
maintaining these services is the responsibility of the remote facility. An advantage to this option is that 
you have local autonomy. 


ik 

Operational Needs 

• Bandwidth to accommodate data, video, voice, etc. 

• Network support/availability 

• Coordination- with POIC/Crew 

• Database population/parameters to monitor 

• Commanding - onboard vs. ground 

• Security 

• Storage 

• Backups 


• Operational costs - personnel/equipment 

• Maintenance costs - HW/SW 

• Interface costs 

• Data archival costs 

• Training 

• Crew Time 


Figure 4- Cost Analysis 

For the user that decides to use services and capabilities of the control center, a cost reduction can be 
realized since the costs of the infrastructure items of the control center are shared by multiple users or 
projects. Control center services can range from shared facility space to providing equipment, interfaces, 
infrastructure services, or personnel for operations and maintenance tasks. 

A user may choose a phased approach to his remote ops concept. Payload users may prefer close 
proximity to controllers operating the real-time system during the initial or critical phases of a payload’s 
operation. This would provide the security of infrastructure personnel in case problems occur. As payload 
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operations become more routine, the use may develop a transition plan from the control center to his 
remote location. 

An efficient and effective solution may be to augment the remote facilities services and capabilities with 
the use of some control center service and capabilities. A cost/benefit analysis needs to be completed 
which weighs operations needs against costs to determine the optimized solution as shown in figure 4. 

Lessons Learned 


Four general categories of lessons learned have been identified through the experiences of the HOSC 
and our remote operations history: 

1 . Ensure an understanding of the onboard system requirements and limitations 

By understanding the onboard system requirements and limitations a user can maximize their payload 
operations concept to take advantage of all onboard capabilities and minimize limitation impacts. 

2. Understand the ground system complexities (i.e., security requirements, IT support requirements, 
network requirements, etc) 

Ensure that the ground system architecture provides easy integration of changing baselines. Ground 
system complexities can come in many flavors, it may be newly mandated security requirements, new 
COTS products integrated into your operations environment, fiscal drivers requiring an evaluation and 
implementation of commodity based systems, etc. 

3. Understand the total end-to-end user needs 

Equal emphasis should be placed on all phases of the mission. From our experience, we initially 
concentrated a large quantity of effort on the “real-time" portion of the mission and did not provide 
enough attention to the pre-mission preparation and post mission analysis phases. 

4. A ground system can never provide a 100% solution to meet all customers’ needs. The goal is to 
provide standard interfaces and user tools that are easy to use and allow customers to develop 
custom solutions for those “unmet” requirements. Each user must understand what limitations apply 
and take advantage of the provided tools. 


Benefits for Future Programs 

Future programs can derive benefits from this paper that can help design, implement, and maintain a cost 

effective and efficient ground infrastructure in support of all mission phases. The items that should be 

taken into consideration when implementing an operations facility include: 

1 . Understanding the existing capabilities and limitations of spacecraft systems and interface facilities 
prior to building your control facility. Realize no solution is perfect. Onboard and ground systems 
have both capabilities and limitations. Having a thorough understanding of these items will help you 
design your payload and ground support facility more cost effectively. 

2. Evaluate existing tools that may be used to satisfy requirements. 

3. Maximize the use of commodity-based systems and open source products. 

4. Realize that a user does not have to be local to a control center to conduct realtime operations 

5. A tiered approach to security minimizes access to crucial systems. 

6. Consider conducting a cost analysis to help derive a “just right” cost effective solution. 
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